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Abstract 

In this paper, we aim to prove the linear rate convergence of the alternating direction 
method of multipliers (ADMM) for solving linearly constrained convex composite optimiza¬ 
tion problems. Under an error bound condition, we establish the global linear rate of con¬ 
vergence for a more general semi-proximal ADMM with the dual steplength being restricted 
to be in (0,(1 + \/5)/2). In our analysis, we assume neither the strong convexity nor the 
strict complementarity except the error bound condition, which holds automatically for con¬ 
vex composite quadratic programming. This semi-proximal ADMM, which covers the classic 
one, has the advantage to resolve the potentially non-solvability issue of the subproblems in 
the classic ADMM and possesses the abilities of handling the multi-block cases efficiently. We 
shall use convex composite quadratic programming and quadratic semi-definite programming 
to demonstrate the significance of the obtained results. Of its own novelty in second-order 
variational analysis, a complete characterization is provided on the isolated calmness for the 
convex semi-definite optimization problem in terms of its second order sufficient optimality 
condition and the strict Robinson constraint qualification for the purpose of proving the 
linear rate convergence of the semi-proximal ADMM when applied to two- and multi-block 
convex quadratic semi-definite programming. 
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1 Introduction 

In this paper, we shall study the linear rate convergence of the alternating direction method of 
multipliers (ADMM) for solving the following convex composite optimization problem 

min {i&(y) + g(y) + <p(z) + h{z) : A*y + B*z = c , y € y, z£ Z}, (1.1) 

where y and Z are two finite-dimensional real Euclidean spaces each equipped with an inner 
product (•,•) and its induced norm || ■ ||, i? : y —>• (—oo,+oo] and ip : Z —>• (—oo,+oo] are 
two proper closed convex functions, g : y —> (—oo,-|-oo) and h : Z —> (—oo,+oo) are two 
continuously differentiable convex functions (e.g., convex quadratic functions), A* : y —» X and 
B* : Z —>• X are the adjoints of the two linear operators A : X —>• A’ and B : X ^ Z, respectively, 
with A being another real finite-dimensional Euclidean space equipped with an inner product 
(•, •) and its induced norm || • || and c € X is a given point. For any convex function 0 : X —>• 
(— 00 , 00 ], we use dorri 0 to define its effective domain, i.e., dorri 0 := {x £ X : 0(x) < 00 }, epi0 
to denote its epigraph, i.e., epi0 := {(x,i) € X X 3ft : d(x) < t} and 9* : X —>• (— 00 , 00 ] to 
represent its Fenchel conjugate, respectively. 

The classic ADMM was designed by Glowinski and Marroco [28] and Gabay and Mercier [25] 
and its construction was much influenced by Rockafellar’s works on proximal point algorithms 
(PPAs) for solving the more general maximal monotone inclusion problems [43, 44]. The readers 
may refer to Glowinski [27] for a note on the historical development of the classic ADMM. The 
convergence analysis for the classic ADMM under certain settings was first conducted by Gabay 
and Mercier [25], Glowinski [26] and Fortin and Glowinski [22]. For a recent survey on this, see 

[19]- 

Our focus of this paper is on the linear rate convergence analysis of the ADMM. This shall 
be conducted under a more convenient semi-proximal ADMM (in short, sPADMM) setting 
proposed by Fazel et al. [21] by allowing the dual step-length to be at least as large as the 
golden ratio of 1.618. This sPADMM, which covers the classic ADMM, has the advantage 
to resolve the potentially non-solvability issue of the subproblems in the classic ADMM. But, 
perhaps more importantly it possesses the abilities of handling multi-block convex optimization 
problems. For example, it has been shown most recently that the sPADMM plays a pivotal role 
in solving multi-block convex composite semi-definite programming problems [49, 35, 10] of a 
low to medium accuracy. We shall come back to this in Section 3. 
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For any self-adjoint positive semi-definite linear operator M. : X —» X, denote ||x||>i := 
(x, Mx) and dist_M(x, C) = inf T / e ^ ||x' — x\\m for any x G X and any set C C X. We use 
X to denote the identity mapping from X to itself. Let er > 0 be a given parameter. Write 
'dg(-) = + and <ph{-) = ¥>(■) +M')- The augmented Lagrangian function of problem (1.1) 

is defined by 

C a (y,z-,x) := tf g (y)+<p h (z)+(x,A*y+B*z-c)+^\\A*y+B*z-c\\ 2 , V( y,z,x) G XxZxX. (1.2) 

Then the sPADMM may be described as follows. 

sPADMM: A semi-proximal alternating direction method of multipliers for solving 
the convex optimization problem (1.1). 

Step 0. Input (y°,z°,x°) € dom d X dom tp X X. Let r G (0,oo) be a positive 
parameter (e.g., r G (0,(1 + y/5)/2)), and S : y —>• y and T : Z — Z be 
two self-adjoint positive semi-definite, not necessarily positive definite, linear 
operators. Set k := 0. 

Step 1. Set 


- fc+1 G argmin C a (y,z k -,x k ) + ^\\y- y k \\ 2 s , 

(1.3a) 

: fc+1 G argmin C a {y k+1 , z-, x k ) + \\\z - z k f T , 

(1.3b) 

X+ 1 = x k + Td(A*y k+1 + B*z k+1 - c). 

(1.3c) 


Step 2. If a termination criterion is not met, set k := k + 1 and go to Step 1. 

The sPADMM scheme (1.3a)-(1.3c) with 5 = 0 and T = 0 is nothing but the classic ADMM 
of Glowinski and Marroco [28] and Gabay and Mercier [25] . When B = X and A is surjective, 
the global convergence of the classic ADMM with any r G (0, (1 + \/5)/2) has been established 
by Glowinski [26] and Fortin and Glowinski [22] . Interestingly, in [24] , Gabay has further shown 
that the classic ADMM with r = 1, under the existence condition of a solution to the Karush- 
Kuhn-Tucker (KKT) system of problem (1.1), is actually equivalent to the Douglas-Rachford 
(DR) splitting method applied to a stationary system to the dual of problem (1.1). Moreover, 
Eckstein and Bertsekas [18] have proven that the DR splitting method can be equivalently 
represented as a special PPA. Thus, one may always use known results on the DR splitting 
method and the PPA to study the properties of the classic ADMM with r = 1 (this does not 
apply to the case that r / 1 of course) though the corresponding transformations can be much 
involved. The above sPADMM scheme (1.3a)-(1.3c) with 5^0 and T A 0 was initiated by 
Eckstein [16] to make the subproblems in (1.3a) and (1.3b) easier to solve. Using essentially the 
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same variational techniques developed by Glowinski [26] and Fortin and Glowinski [22], Fazel 
et al. developed an extremely easy-to-use convergence theorem for the sPADMM [21, Appendix 
B] when the dual step-length r is chosen to be in (0, (1 + \/5)/2). In [46], Shefi and Teboulle 
conducted a comprehensive study on the iteration complexities, in particular in the ergodic 
sense, for the sPADMM with r = 1 and B = T. Related results for the more general cases can 
be found, e.g., in [33] for the case that the linear operators S and T are allowed to be indefinite 
and in [11] for the case that the objective function is allowed to have a coupled smooth term. 
For details on choosing S and T, one may refer to the recent PhD thesis of Li [34]. 

Compared with the large amount of literature 1 mainly being devoted to the applications of 
the ADMM, there is a much smaller number of papers targeting the linear rate convergence anal¬ 
ysis though there do exist a number of classic results and several interesting new advancements 
on the latter. By using the aforementioned connections among the DR splitting method, PPAs, 
and the classic ADMM with r = 1, we can derive the corresponding linear rate convergence of 
the ADMM from the works of Lions and Mercier [36] on the DR splitting method with a globally 
Lipschitz continuous and strongly monotone operator and Rockafellar [43, 44] and Luque [37] on 
the convergence rates of the PPAs under various error bound conditions imposed on the inverse 
of maximal monotone operators. For example, within this spirit, Eckstein and Bertsekas [17] 
proved the global linear convergence rate of the ADMM with r = 1 when it is applied to linear 
programming by using the equivalence of the ADMM and a PPA. For recent new developments 
on the linear convergence rate of the ADMM, we can roughly categorize them into the following 
three cases: 

(i) For convex quadratic programming, Boley [2] provided a local linear convergence result for 
the ADMM with r = 1 under the conditions of the uniqueness of the optimal solutions to 
both the primal and dual problems and the strict complementarity; in [29], Han and Yuan 
removed the restrictive conditions imposed by Boley and established the local linear rate 
convergence of the generalized ADMM in the sense of Eckstein and Bertsekas [18] for the 
subsequence {(z k ,x k )}; and in [50], Yang and Han showed that the local linear rate result 
in [29] can be globalized under a slightly more general setting for the ADMM with r = 1 
and a linearized ADMM (a special case of sPADMM with 5^0 and T 0) with r = 1, 
where for the latter the linear rate is established for the whole sequence {(y k ,z k ,x k )} 
instead of only the subsequence {( z k ,x k )}. We remark that when either S >- 0 or T >- 0 
fails to hold, the linear rate convergence analysis in [50] is no longer valid. 

1 For example, according to Google Scholar, the survey paper by Boyd et al. [7] on the applications of the 
ADMM with r = 1 has been cited more than 2,289 times as of August 2, 2015. 
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(ii) In [12], Deng and Yin provided a number of scenarios on the linear rate convergence for 

the ADMM and sPADMM with r = 1 under the assumption that either or <Ph( m ) 

is strongly convex with a Lipschitz continuous gradient in addition to the boundedness 
condition on the generated iteration sequence and others. Deng and Yin’s focus is mainly 
on problems being reformulated from unconstrained composite models with applications 
in sparse optimization, e.g., the models of Lasso regularized with strongly convex terms. 
They also made a detailed comparison between their most notable linear rate convergence 
result and that of Lions and Mercier [36] on the DR splitting method when applied to a 
stationary system to the dual of problem (1.1). 

(iii) Assuming an error bound condition and some others, Hong and Luo [30] provided a lin¬ 
ear rate convergence of the multi-block ADMM with a sufficiently small step-length r. 
Theoretically, this constitutes important progress on understanding the convergence and 
the linear rate of convergence of the ADMM. Computationally, however, this is far from 
being satisfactory as in practical implementations one always prefers a larger step-length 
for achieving numerical efficiency. 

In this paper, we aim to resolve the linear rate convergence issue for the sPADMM scheme 
(1.3a)-(1.3c) with r € (0, (1 + y/h)/2) assuming neither the strong convexity for 'dg(-) or <Ph( m ) 
nor the strict complementarity. Special attention shall be paid to convex composite quadratic 
programming and quadratic semi-definite programming. For the former, we have a complete 
picture and for the latter we show how far we have progressed. More specifically, our main 
contributions made in this paper include but are not limited to: 

(1) Under an error bound condition only, we provide a very general linear rate convergence 

analysis for the sPADMM with r € (0, (1 + V5)/2). This is made possible by construct¬ 
ing an elegant inequality on the iteration sequence via re-organizing the relevant results 
developed in [21, Appendix B], 

(2) For convex composite quadratic programming, the global linear convergence rate is obtained 

with no additional conditions as the error bound assumption holds automatically. By 
choosing the positive semi-definite linear operators S and T properly, in particular T = 0, 
we demonstrate how the established global linear rate convergence of the sPADMM can 
be applied to multi-block convex composite quadratic conic programming. 

(3) For convex composite quadratic semi-definite programming (SDP), a linear convergence rate 

is established under the assumption that both the primal and dual problems satisfy the 
second order sufficient optimality condition, one of eight equivalent conditions proven in 
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this paper. This is achieved via characterizing the isolated calmness of the corresponding 
optimality systems. 

(4) The obtained results on the isolated calmness for convex and non-convex semi-definite 
optimization problems are not only important for the linear rate convergence analysis 
of the sPADMM but also are interesting in their own right in the context of sensitivity 
analysis for optimization problems with non-polyhedral cone constraints. 

The remaining parts of the this paper are organized as follows. In Section 2, we conduct 
brief discussions on the optimality conditions for problem (1.1) and on both the calmness and 
isolated calmness for multi-valued mappings. Section 3 is divided into three parts with the first 
part focusing on deriving a particularly useful inequality for the iteration sequence generated 
from the sPADMM. This inequality, which grows out of the results in [21, Appendix B], is then 
employed to build up a general linear rate convergence theorem under an error bound condition. 
The third part of this section is about the applications of the linear convergence theorem of the 
sPADMM to important convex composite quadratic conic programming. Section 4 is devoted to 
the characterization of the isolated calmness for composite semi-definite optimization problems, 
which are not necessarily convex. The sufficient conditions for non-convex semi-definite opti¬ 
mization problems, which are strongly motivated by the work done in [47] on Robinson’s strong 
regularity, can be regarded as natural extensions to those established by Zhang and Zhang [51]. 
The complete characterization of the isolated calmness in the convex case represents a significant 
step forward in second order variational analysis on convex optimization problems constrained 
with non-polyhedral convex cones. In Section 5, for convex composite quadratic semi-definite 
programming, we provide further deep results on the isolated calmness by relating the second 
order sufficient optimality condition for the primal problem equivalently to the strict Robinson 
constraint qualification for the corresponding dual problem. We make our final conclusions in 
Section 6. 

2 Preliminaries 

In this section, we summarize some useful preliminaries for subsequent analysis. 

2.1 Optimality conditions 

For a multifunction F : y ^ y, we say that F is monotone if 

(y'-y,C'-0> o, V£' g F( y ’), e F( y ). (2.1) 
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It is well known that for any proper closed convex function 9 : X —>• (— 00 , 00 ], 89(-) is a monotone 
multi-valued function (see [42]), that is, for any w\ € dom 9 and any u >2 € dom 9, 

(£ — (,wi — w 2 ) > 0 , V£ € d9(wi), VC G 89(w 2 ). ( 2 . 2 ) 

In our analysis, we shall often use the optimality conditions for problem (1.1). Let (y,z) € 
dom('d) x dom(</?) be an optimal solution to problem (1.1). If there exists x € X such that 
(y, z, x) satisfies the following KKT system 

0 € d‘d(y) + Vy(y) + Ax, 

< 0 € 8<p(z) + Vh(z) + Bx, (2-3) 

c-A*y-B*z = 0, 

then ( y,z,x ) is called a KKT point for problem (1.1). Denote the solution set to the KKT 
system (2.3) by Q. The existence of such KKT points can be guaranteed if a certain constraint 
qualification such as the Slater condition holds: 

3 (?/, z') € ri (dorn('d) x dom( 95 )) n {(y, z) € y x Z : A*y + B*z = c}, 

where ri(S) denotes the relative interior of a given convex set S. In this paper, instead of using 
an explicit constraint qualification, we make the following blanket assumption on the existence 
of a KKT point. 

Assumption 2.1. The KKT system (2.3) has a non-empty solution set. 


Denote u := (y, z, x) for y € y, z € Z and x € X. Let U := y x Z x X. Define the KKT 
mapping R : U —>• U as 


R(u) 


^ V ~ PL? [y ~ (Vy(y) + Ax)} ^ 
z - Pr^fz - (Wh(z) + Bx)} , 
^ c — A*y — B*z 1 


Vu£U, 


(2.4) 


where for any convex function 9 : X —>• (— 00 , 00 ], Prg(-) denotes its associated Moreau-Yosida 
proximal mapping. If 9{) = the indicator function over the closed convex set /CCA, 

then Pi'e(-) = n^;(-), the metric projection operator over 1C. Then, since the Moreau-Yosida 
proximal mappings Pr^(-) and Pr ¥ ,(-) are both globally Lipschitz continuous with modulus one, 
the mapping R(-) is at least continuous on U and 


Vm 


R(u) = 0 m£!1. 
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2.2 Calmness and isolated calmness 


Let X and y be two finite-dimensional real Euclidean spaces and F : X y be a set-valued 
mapping with ( x°,y° ) € gphT 1 , the graph of F. Let By denote the unit ball in y. 

Definition 2.1. The multi-valued mapping F : X ^ y is said to be calm at x° if there is a 
constant ko > 0 along with a neighborhood V of x° such that 

F(x) C F(x°) + Ko||a; — x°|| By, Vi 6 L 

The above definition of calmness for the multi-valued mapping F comes from [45, 9(30)] and 
it was called the upper Lipschitz continuity in [40]. Recall that the multi-valued mapping F is 
called piecewise polyhedral if gph F is the union of finitely many polyhedral sets. In one of his 
landmark papers, Robinson [41] established the following important property on the calmness 
for a piecewise polyhedral multi-valued mapping. 

Proposition 2.1. If the multi-valued mapping F : X ^ y is piecewise polyhedral, then F is 
calm at x°. 

Next, we give the definition of isolated calmness for F : X y at x° for y°. 

Definition 2.2. The multi-valued mapping F : X ^ y is said to be isolated calm at x° for y° if 
there is a constant hq > 0 along with a neighborhood V of x° and a neighborhood W of y° such 
that 

F{x)nW C{y°} + K 0 \\x-x°\\By, Mx^V. 

The isolated calmness given in Definition 2.2 was called differently in the literature, e.g., the 
local upper Lipschitz continuity in [13, 32], to distinguish it from Robinson’s definition of upper 
Lipschitz continuity [40]. Here we adopt the usage in [15, 8]. The concept of graphical derivative 
of F [45, 8.33 Definition] is a convenient tool for investigating the isolated calmness property. 
The graphical derivative of F at x° for y° e F(x°) is the set-valued mapping DF(x°\y°) \ X =%y 
whose graph is the tangent cone T^p^x 0 , y°), namely for any (u,v) € X x y, 

v € DF(x°\y°)(u) 4=> (u,v) € T gphF (x°, y°). 

In other words, v € DF(x°\y°)(u) if and only if 

' i j 

there exist sequences t,f c —>• 0 +, u‘ K —> u and v K v 

' such that 6 F(X ° + tkuk) - V ° for all k. 

< tk 
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It follows from [45, 8(19)] that the following equivalence holds: 

v £ DF(x°\y°)(u ) u £ D(F~ 1 )(y°\x°)(v). (2.5) 

A basic characterization of the isolated calmness property for a set-valued mapping at a 
point is given by the following lemma. 

Lemma 2.1. (King and Rockafellar [31], Levy [32]) Let ( x°,y° ) £ gphi 7 . Then F is isolated 
calm at x° for y° if and only if {0} = HF(x°|r/°)(0). 

3 A general theorem on the linear rate convergence 

In this section, we shall establish a general theorem on the linear convergence rate of the 
sPADMM scheme (1.3a)-(1.3c). 

First we recall the global convergence of the sPADMM from [21, Appendix B]. Since both dd 
and dip are maximally monotone, there exist two self-adjoint and positive semi-definite linear 
operators and H Vh such that for all y',y £ dom$ 5 , £ £ dd g (y) and £' £ dd g (y'), and for all 
z',z £ dom ip h , £ £ dip h (z) and £' € dip h (z'), 

<£' -u-y)> II v' - y\\h g , (C - C ,J-z)> Ik' - 4l Vh • (3.i) 

For notational convenience, \et £ \ X ^ U \= y x Z x X be a linear operator such that its 
adjoint £* satisfies £*(y, z, x) = A*y+B*z for any ( y,z,x ) £ y x Z x X and for u := (y, z,x) £U 
and v! := (■ y' , z', x') £ U , define 

6{u,u') := (rcr)' 1 |k ~ x '\\ 2 + \\v ~ v'Ws + Ik - z '\\t + a \\B*( z ~ z ')\\ 2 ■ 

The following theorem, which will be used in the following, is adapted from Appendix B of [21]. 

Theorem 3.1. Let Assumption 2.1 be satisfied. Suppose that the sPADMM generates a well 
defined infinite sequence {u k }. Let u = ( y,z,x ) £ £l. For k> 1, denote 

[ 5k := t( 1 — r + minjr, r~ 1 })a\\B*(z k — z fc-1 )|| 2 + \\z k — z^ 1 1|2-, 

< (3.2) 

( v k :=S k + \\y k ~y k 1 |ll + 2 || 2 / fc - 2 /|||^ + 2\\z k - z\\l^. 

Then, the following results hold: 
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(i) For any k > 1, 

d(u k+1 ,u ) + || z k+l - z k \\ 2 T + (1 - min{r,r- 1 })a||rC/' +1 ,^ +1 ,0) - c|| 2 
- 9{u k , u) + || z k - z k ~ x \\ 2 T + (1 - minjr, t ~ 1 }) o \\ 8 * (y k , z k , 0) - c|| 2 


(3.3) 


< - 


u k+1 + (1 


r + min{r, (y k+1 , z k+1 ,0 ) - c|| 2 


(ii) Assume that both T,$ g + S + aAA* and Z v , h + T + aBB* are positive definite so that the 
sequence {u k } is automatically well defined. Ifr£ (0, (l + \/5)/2), then the whole sequence 
{(y k ,z k ,x k )} converges to a KKT point in Ft. 


For any self-adjoint linear operator M. : X —>• X, we use A max (.M) to denote its largest 
eigen-value. Define k := max{/ci, ^ 3 }, where 

«1 := 3||<S||, k 2 := max{3aA max (A4*), 2||T||} 

and 

k 3 := 3(1 - r) 2 <7A max (A4*) + 2(1 - r) 2 aX m ^(BB*) + a~\ 

Let 

Ho := k Diag (S,T + oBB\ (tV)" 1 !) (3.4) 

be a block-diagonal positive semi-definite linear operator from y x Z x A to itself such that 

Ho(y, z,x) = k ( Sy , (T + aBB*)z , (t 2 <t) -1 x) , M (y , z, x) £ y x Z x X . 

Lemma 3.1. Let {u k := (y k , z k , x k )} be the infinite sequence generated by the sPADMM scheme 
(1.3a)-(1.3c). Then for any k > 0, 

\\u k+1 - u k \\ 2 Uo > \\R(u k+1 )\\ 2 - (3-5) 

Proof. The optimality condition for (1.3a) is 

0 G dd{y k+1 ) + Vg(y k+1 ) + A[x k + a(A*y k+1 + B*z k - c)] + S{y k+1 - y k ). (3.6) 
From the definition of x k+1 , we have 

x k + a(A*y k+1 + B*z k - c) = -aB*(z k+1 - z k ) + x k + T~\x k+1 - x k ). 

It then follows from (3.6) that 

0 G d , d(y k+1 ) + Vg(y k+1 ) + A[x k + a(A*y k+1 + B*z k — c)] + S(y k+1 — y k ) 

= dd(y k+1 ) + Vg(y k+1 ) + A[aB*(z k - z k+1 ) + x k + T~\x k+l - x k )\ + S(y k+1 - y k ), 
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which implies 


yk-\-l __ 

Pr,j ( y k+1 — ( Vg(y k+1 ) + A[aB*{z k — z k+1 ) + x k + r“ 1 (x fc+1 — x k )\ + S(y k+1 — y k ))) . 

(3.7) 

Noting that since z k+1 is a solution to the subproblem (1.3b), we have that 

0 € d<p(z k+1 ) + \7h(z k+1 ) + Bx k + aB(A*y k+1 + B*z k+1 - c) + T{z k+1 - z k ), 
which is equivalent to 

0 € dtp(z k+1 ) + Vh(z k+1 ) + B[x k + T~\x k+1 - x k )] + T{z k+1 - z k ). 

Thus, we have 

z k+1 = Vi {p [z k+l - (Wh(z k+1 ) + B[x k + T~\x k+1 -x k )]+T{z k+1 - z k )^ . (3.8) 

Note that from (1.3c), 

x k+1 = x k + rcr(A*y k+1 + B*z k+1 - c ). (3.9) 

Then, by coming (3.7), (3.8) and (3.9) and noticing of the Lipschitz continuity of the Moreau- 
Yosida proximal mappings, we obtain from the definition of R (-) in (2.4) that 

|| R ( u ‘+ 1 )|| 2 

< || S(y k+1 — y k ) + aAB*(z k+1 — z k ) + (1 — r _1 )„4,(;r fc+1 — x fc )|| 2 
+ || T(z k+1 — z k ) + (1 — T~ l )B{x k+l — x fc )|| 2 + (r(j)~ 2 \\x k+l — x k \\ 2 

< [3||«S|| \\y k+1 - y k ||| + 3cr 2 A max (+l+l*)||i5(^ A:+1 - z k )\\ 2 + 3(1 - r _1 ) 2 ||„4(x fc+1 - x k )\\ 2 ] 

+ [2||T||||z fc+1 — z k \\j- + 2(1 — r -1 ) 2 ||£>(;r fc+1 — x k )\\ 2 + ( Ta)~ 2 \\x k+l — x fc || 2 ] 

< ki|| y k+1 - y k ||| + /c 2 || z k+1 - z k Wr+aBB* + K 3 (r 2 cr) _1 ||a; fc+1 - x k \\ 2 , 

which immediately implies (3.5). □ 

For any r G (0, oo), define 

5 — t — 3 min{T, r 1 } p + 1 — r + min{r, r -1 } 

4 8 

Note that 

1/4 < s T < (5 - 2>/3)/4 & 0 < t r < 1/8, Vr G (0, (1 + V5)/2). (3.10) 

Denote 

M := Diag (5 + E^T + E^ + aBB*, (ra)- 1 !) + s T aSS* (3.11) 
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and 


LL := Diag ( S + ^E^, T + ^E^ + roBB*, At T (r 2 a) l l ) + t r a££*. 


Then we immediately get the following relation 

kLL h min{r,4t T }^o + nt T a££*, Vr € ( 0 , (1 + \/5)/2). 
Proposition 3.1. Let t E (0, (1 + v / 5)/2). T/ien 

S,? g + S + <tA 4* >- 0 & E Vh + T + aBB* yO^MyO^LLyO. 


(3.12) 


(3.13) 


Proof. Since, in view of (3.10), it is obvious that M 0 -<=>- LL y 0, we only need to show that 


E^ g + S + aAA* y 0 & E^ + T+aBB* y 0 M >- 0. 

First, we show that Y,$ g + 5 + aAA* y 0 & E^ + T + ( 7 £>£>* >- 0 ==y M y 0 . Suppose that 
Y,$ g + S + aAA* y 0 & Yi^ h + T + <x£>£>* >- 0 , but there exists a vector 0 7 - d := (d y , d z ,d x ) € 
y x Z x X such that (d, Aid) = 0. By using the definition of M and (3.10), we have 


d x — 0, (+ T + aBB*)d z — 0, (E$ g + S)y — 0 &: £*(dy , d z , 0) — 0, 

which, together with the assumption that E$ g + S + aAA* y 0 &; E^ + T + aBB* y 0 , imply 
d = 0. This contradiction shows that My 0. 

Next, suppose that My 0. Since s T > 0 and for any d = (0, d z , 0) E y x Z x X , (d, Md) = 
(d z , (Ep h + T + (1 + s r )aBB*)d z ), we know that E^ + T + cr£>£>* >~ 0 . Similarly, since for any 
d = (d y , 0, 0) E y x Z x X , ( d,Md ) = (dy, (S^ + 5 + s T cr.A.4*)dy), we know that E^ g + S + 
<r AA* y 0 . So the proof is completed. □ 

Proposition 3.2. Let t E (0, (1 + \/5)/2) and {(y k , z k , x k )} be an infinite sequence generated 
by the sPADMM. Then for any u = (y, z, x) E f i and any k > 1, 

\\u k+1 - u\\m + \\z k+1 - A\t < (\\u k - «||M + \\z k - 2 * _1 ||r) - lh fc+1 - (3.1.4) 

Consequently, we have for all k > 1, 

dist^(« fc+ 1 ,12) + ||^ fc+1 — 2 r fc H 7 - < ^dist^j(u fc , Ll) + ||z fc — z k ~ 1 \\q- S j — \\u k+1 — u k ||^. (3.15) 

Proof. Let u = ( y,z,x ) € 12 be fixed but arbitrarily chosen. From part (i) of Theorem 3.1, we 


12 


have for k > 1 that 


(rcr) 1 ||x fc+1 — x\\ 2 + || y k+l — y||| + ||z fc+1 — z\\j- + a\\B*(z k+1 — z)|| 2 
+ ||z fc+1 _ z k ||2_ + _ minj^r- 1 }) a\\£*(y k+1 , z k+1 , 0) - c|| 2 

< (tct) -1 ||.T fc - x|| 2 + II y k - y\\l + \\z k - zf T + a\\B*(z k - z )|| 2 
+ || 2 : fc — z k ~ 1 \\j- + (l — min{r, r -1 }) a\\£*(y k , z k , 0) — c|| 2 
- {g[t - r 2 + rmin{r,r- 1 }]||^(z fc + 1 - z k )f 
+ ||z fc+1 - z k II 2 - + \\y k+1 - y k ||| + 2|| y k+1 - y|||^ + 2||++ 1 - zf^ 

+ (1 — r + min{r, r _1 })cr||<?*(y A:+1 , z k+1 , 0) — c|| 2 } . 

By reorganizing the terms in (3.16), we obtain 


(rcr) 1 \\x k+1 — x || 2 + \\y k+1 — 2 /||;| + ||z fc+1 — z\\j- + cr||£>*( 2: fc+1 — z, 0)|| 2 
+ \\z k+l — z k \\j- + \ (5 — r — 3min{r, r^ 1 }) a\\£*(y k+1 , z k+1 , 0) — c || 2 

+\\y k+1 - y\\%, + \\ zk+1 - z\\e v 

< (Tcr) _ 1 [|a: fc — x|| 2 + \\y k — y\\g + ||z fc — z\\j- + cr\\J3*(z k — z )|| 2 

H-1|— z fc_ 1 ||j- + \ (5 — r — 3min{r, r^ 1 }) o\\£*(y k , z k , 0) — c || 2 

+\\v k - y\\l„ + \\z k - z\\l^ 

- {f 7 r||fi *(^ +1 - z fc )|| 2 + || z k+1 - z k f T + \\y k+1 - y k \\ 2 s 

+l|y fc+1 - yWl# + II y k - Vlll# + \\ zk+1 ~ z \\l v + \\ zk ~ z \\% 

+ 5(1 — t + min{r, T~ 1 })cr||£*(y fc + 1 , z k+1 , 0 ) — c || 2 

+|(1 - r + min{r, r _ 1 })cr[||f*( 2 / fc+ 1 , 2 ; fc+ 1 , 0 ) - c || 2 + \\£*(y k , z k , 0 ) - c|| 2 ]} 
or equivalently 


(tct) 1 ||x A:+1 — m || 2 + \\y k+1 — y||| + ||2; fe+1 — z\\j- + a\\B*(z k+1 — z) || 2 
+ || 2; A:+1 - z k \\j- + s T a\\£*(y k+1 , z k+1 , 0 ) - c || 2 

fc+1 -y\\l t 

-liijfc =112 


1 r 

+l|y fc+ 1 -y|l!*+ ll^ + 1 -^ll! 


I h 

\X — X\ 


+ 


+ \\z k -z\\ 2 T + a\\B*{z k -z) 


t + s r a \\ £ *(y > z ’°) - c ll + lly K - s/IIL +ll*“ - z|| E „ 


< (to-) 

+ 11 z k - z 

- {o-r||, 8 *( 2: fe+1 - z k )|| 2 + ||a ; fc+1 - z fc ||^- + || y k+l 

+l|y fc+1 - y\\% + II y k - y|||„ + I \z k+1 - z\\l 
+ 5 (l — r + min{r,T -1 }) cr\\£*{y k+l , z k+1 , 0 ) - c|| 


- 7/ fc II 2 

y iis 

+ | \z k — z\ 


+1(1 - T + min{r,T- 1 }) ( 7[||^(2/ fc + 1 ,^+ 1 ) 0) - c|| 2 + \\£*{y k , z k , 0) - c 
Using equalities 


£* ( y k+1 , z k+1 , 0) - c = A* (y k+1 - y) + B* ( z k+l - z) , 
£* {y k , z k , 0) - c = A* (y k - y) + B*{z k - z), 

£*(y k+1 , z k+1 , 0 ) — c = (ra)~ l {x k+1 — x k ) 


(3.16) 


(3.17) 
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and inequalities 


II y k+1 - v IIe* s + II y k ~ y\\h g - \h k+1 

\\z k+l — z\\y + \\z k — z\\y > A|| Z k+l 

£j(p h Z 

\\£*(y k+ 1 ,z k+ 1 , 0 ) - c || 2 + || £*(y k , z k , 0 ) 

we obtain from (3.17) and the definitions of s T and t T that 

(rcr)~ 1 ||x fc+1 — x || 2 + || y k+1 — y\\s + ||z fc+1 — z\\j- + <r|| B*(z k+1 — z) || 2 
+ ||^ fc+1 — z k \\'j- + s T cr\\A* (y k+1 — y) + B*(z k+1 — z) || 2 

+l|y fc+1 - y\\l# g + \\z k+1 - z\\% h 

< (T(j)~ l \\x k — x|| 2 + ||y fc — y||| + || z k — z\\j- + (j\\B*(z k — z)|| 2 

+ ||z fc - z k ~ l f T + s T a\\A*(y k - y) + B*(z k - z )|| 2 + ||/ - y|| 2 ^ + ||z fc - z|| 2 ^ 

— {crr||B*(z fc+1 - z fe )|| 2 + ||z fc+1 - z k \\j-+ ||y fc+1 - y k \\s + \\\y k+1 - y k ||| # 

+ 5 II z k+1 - z k |||^ + 4t T (r 2 cr) _ 1 ||a ; fc+1 - x k \\ 2 
+t r a\\A* (y k+1 — y k ) + B*(z k+1 — z fc )|| 2 }, 

which shows that (3.14) holds. By noting that fl is a nonempty closed convex set and (3.14) 
holds for any u € 14, we immediately get (3.15). □ 


y 1 is,, 


— z 


— C 


k ||2 


-‘Vh ' 


> i||i*(j / H1 - y k ) + B*{z k+1 - z k 


For establishing the linear rate of convergence of the sPADMM, we need the following error 
bound condition. 

Assumption 3.1 (Error bound condition). For any given u € 17, there exist positive constants 
5 and y > 0 such that 


dist(n, 14) < r/||i?(rt)||, Vtt€{tt€77: ||it — u|| < 5}. 


(3.18) 


Theorem 3.2. Let t € (0,(1 + v/5)/2). Suppose that Assumptions 2.1 and 3.1 hold. Assume 
also that both T,^ g + S + aAA* and T, Vh + T + aBB* are positive definite. Let {(y k ,z k , x k )} be 
the infinite sequence generated from the sPADMM. Then for all k sufficiently large, 


dist z M (u k+1 ,n) + \\z k+l - z k \\ 2 T < y dist^(n fc , 0 ) + ||z* - * 


fc+i 


112 


k -1 M 2 

It 


(3.19) 


where 


/a := (1 + 2 ^ 4 ) 1 (1 T K 4 ) < 1 &: K 4 := minjr, 4f T } (r/ 2 KA max (A4)) 1 > 0. 

Moreover, there exists a positive number ? € [y, 1) such that for all k > 1, 


dist)h((w fc+1 , 14) + ||z ft+1 — z k \\q- < ? dist^ /( (rt fc , 14) + ||^ fc — z K l \\j- 


fc+i 


k ||2 


fc—1112 


(3.20) 
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Proof. From Theorem 3.1 we know that the whole sequence {(y k ,z k ,x k )} generated by the 
sPADMM converges to a KKT point in LI, say u = (y,z,x). Combining Assumption 3.1 with 
Lemma 3.1 we know that there exists a constant tj > 0 that for all k sufficiently large, 

dist 2 (u k+ 1 ,Ti) < r/ 2 ||i?(u fe+1 )|| 2 < 7 j 2 \\u k - u fc+1 ||^ 0 . (3.21) 


From the definition of "H, we have for all k > 0, 

\\z k+1 - z k Wj- < || u k+1 


— u 


k ||2 

un¬ 


it follows from (3.13) and (3.21) that for all k sufficiently large, 


,k+l 


— u k \\y_ > min{r, 4t r }/« 


-in„fc+i 


u ' — u 


k ||2 


iWo 


(3.22) 


> min{r, 41 t }k 1 rj 2 dist 2 (u fc+ 1 , 11 ) > ^dist)^ (u fc+ 1 , Ll). 

Let K 5 = (1 + ac 4 ) _1 . From (3.15) in Proposition 3.2 and (3.22), we have for all k sufficiently 
large that 


dist^ (u k+1 , fi) + \\z k+1 - z k II 2 - - (dist^,(u fc ,L!) + \\z k - z k ~ x f T ) 

< — ((1 — « 5 )|| u k+l — + K 5 H u k+1 — u k \\h) (3.23) 

< — ((1 — K<r>)\\z k+1 - Z k \\r + «5«4dist^(u fc+1 , fl)) . 

Then we obtain from (3.23) that for all k sufficiently large, 

(1 + K 5 K 4 )dist^ /( (r 4 fc+1 , Ll) + (2 - re 5 )|| z k+l - z k \\j- < dist 2 M (u k ,Ll) + || z k - z k ~ l \\j-. 

By noting that 1 + K 5 K 4 = 2 — we obtain the estimate (3.19). 

By combining (3.19) with Lemma 3.1, (3.13) and (3.15) in Proposition 3.2, we can obtain 
directly that there exists a positive number g € [//, 1) such that (3.20) holds for all k > 1. The 
proof is completed. □ 


Theorem 3.2 provides a very general result on the linear rate of convergence for the sPADMM 
under a fairly mild error bound assumption, which holds automatically if R~ l is piecewise 
polyhedral. Since R is piecewise polyhedral if and only if R itself is piecewise polyhedral, we 
obtain the following directly from Theorem 3.2, Proposition 2.1 and Lemma 3.1. 


Corollary 3.1. Let r E (0, (1 + \/5)/2). Suppose that Ll A 0 and that both + S + aAA* 
and Ti Vh + T + crBB* are positive definite. Assume that the mapping R : U —>• U is piecewise 
polyhedral. Then there exists a constant g € (0,1) such that the infinite sequence {(y k ,z k ,x k )} 
generated from the sPADMM satisfies 


dist^ (u k+1 , fl) + ||z 


fc+i 


— z 


Hr ^ ? 


dist)^ (u fc , Ll) + || z k 


— z 


jfe-ii 


V k > 1. 


(3.24) 
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3.1 Applications to convex composite quadratic conic programming 


In this subsection we shall demonstrate how the just established linear rate convergence theorem 
can be applied to the following convex composite quadratic conic programming 


min -(x, Qx) + (c, x) + <p(x) 
s.t. Ax = b, x € 1C, 


(3.25) 


where c E A, b € ;R m , Q : A —>• X is a self-adjoint positive semi-definite linear operator, 
A : A —» is a linear operator, K. is a closed convex cone in X and <f> : X € (— 00 , 00 ] is a 
proper closed convex function whose epigraph is convex polyhedral, i.e., 4 > is a closed proper 
convex polyhedral function. If /C is a polyhedral cone, problem (3.25) is called the convex 
composite quadratic programming (QP). 

By introducing an additional variable u € X, we can rewrite problem (3.25) equivalently as 

min - ( x , Qx) + (c, x) + 5 K (x) + </>(u) ^ 

s.t. Ax = b, x — u = 0. 


Obviously, problem (3.26) is in the form of (1.1). Let the polar of K, be defined by K.° := {d € 
X : (d, x) < 0, Vs E 1C}. Denote the dual cone of /C by K* := — JC°. The Lagrange dual of 
problem (3.26) takes the form of 

max inf < -(x, Qx) + (v, x) 1 + (b , y) - 4>*(—z ) 

ISA’ [2 J 

s.t. s + A*y + v + z = c, s € /C*, 


which is equivalent to 

min 8 k *{s) - (b,y) + ~{w,Qw) +4>*{~z) ^ 

s.t. s + A*y — Qw + z = c, w E W, 

where VV is any linear subspace in A containing Range Q, the range space of Q, e.g., W = 
A or W = Range Q. When W = A, problem (3.27) is better known as the Wolfe dual to 
problem (3.26) (see Fujiwara, Han and Mangasarian [23] for discussions on the Wolfe dual of 
conventional nonlinear programming and Qi [39] on nonlinear semi-definite programming). So 
when Range Q C W A A, one may call problem (3.27) the restricted Wolfe dual to problem 
(3.26). One particularly useful case is the restricted Wolfe dual with W = Range Q. The dual 
problem (3.27) has four natural variable-blocks and can be written in the form of (1.1) in several 
different ways. The cases that we are interested in applying the sPADMM are: 1) if 1C A A, 
then (s, y, w) is treated as one variable-block and 2 the other block; and 2) if JC = A, then (w, y) 
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is treated as one variable-block and s the other block. We shall only discuss case 1) as case 2) 
can be done similarly in a simpler manner. 


First, we consider the application of the sPADMM to the primal problem (3.26). The 
augmented Lagrangian function for problem (3.26) is defined as follows 


£a(x,u;y,z) 


~{x, Qx) + (c, x) + S/c(x) + 4>(u) + {y,b- Ax) + (z, u - x) 

+ ^(||6 — Ax \\ 2 + ||u — x|| 2 ), V (x, u,y, z) € X x X x x X. 


sPADMM: A semi-proxinral alternating direction method of multipliers for solving 
the convex optimization problem (3.26). 

Step 0. Input (x°, u°, y°, z°) € 1C x dom (<f>) x x X. Let r € (0, oo) be a 
positive parameter (e.g., r € (0,(1 + \/5)/2)). Define S : X —>• X to be any 
self-adjoint positive semi-definite linear operator, e.g., S := 0 if JC = X and 
5 := A max (Q + a A* A) X- {Q + a A* A) if K ± X. Set k := 0. 

Step 1 . Set 

x k+1 = argmin C„(x, u k ;y k , z k ) + ^\\x — x k \\% , 
u k+l = argmin C^(x k+1 , tx; y k , z k ), 

yk+i =y k + r(j(6 _ Ax k+ lj & /+i = z k + rc7 ( u fe+i _ x fc+i). 

Step 2. If a termination criterion is not met, set k := k + 1 and go to Step 1. 

It is easy to see from Theorem 3.2 that as long as Assumptions 2.1 and 3.1 for problem (3.26) 
hold and r € (0, (1 + \/5)/2), the infinite sequence {( x k , u k ,y k , z k )} generated by the sPADMM 
for solving problem (3.26) converges to a KKT point of problem (3.26) globally at a linear rate. 
Note that Assumption 3.1 holds automatically if /C is convex polyhedral, e.g., /C = X or K, = . 

Next, we turn to the dual problem (3.27). As mentioned earlier, problem (3.27) has four 
natural variable-blocks. Since the directly extended ADMM to the multi-block case may be 
divergent even the dual setp-length r is taken to be as small as 10 -8 [9], one needs new ideas 
to deal with problem (3.27). Here, we will adopt the smart symmetric Gauss-Seidel (sGS) 
technique invented by Li et al. [35]. For details on the sGS technique, see [34], Most recent 
research has shown that it is much more efficient to solve the dual problem (3.27) rather than 
its primal counterpart (3.26) in the context of semi-definite programming and convex quadratic 
semi-definite programming [49, 35, 34, 10]. At the first glance, this seems to be counter-intuitive 
as problem (3.27) looks much more complicated than the primal problem (3.26). The key point 


17 




for the more efficiency in dealing with the dual problem is to intelligently combine the above 
mentioned sGS technique with the sPADMM, which will be shown below. 

The augmented Lagrangian function C® for problem (3.27) is defined as follows 

£?(s,y,w,z-,x) := 5 K *(s) - (b,y) + \{w,Qw) + (f>*{-z) + (x,s + A*y - Qw + z - c) 

T~ ||s + A*y — Qw + z — c\\ 2 , V (s, y, w, z, x) G X x xWxA’xf'. 

sGS-sPADMM: A symmetric Gauss-Seidel based semi-proximal alternating direction 
method of multipliers for solving problem (3.27). 

Step 0. Input (s°, y°, w°, z°,x°) G JC* x x W x (—dom (jf) x X. Let r G (0, oo) 
be a positive parameter (e.g., r G (0, (1 + \/5)/2)). Choose any two self-adjoint 
positive semi-definite linear operators <Si : —X and £2 : W —>• W satisfying 

<Si + aAA* >- 0 and So + Q + crQ 2 >- 0. Set k := 0. 

Step 1 . Set 

w k+ ^ = argmin C^(s k ,y k ,w,z k ;x k ) + ^\\w — w k ||| 2 , 
y k+ \ = argmin £®(s k ,y,w k+ ^,z k -,x k ) + \\\y ~ y k \\s 1 , 
s k+1 = argmin (s, y k+ X w k+ X z k \x k ), 

V k+1 = argmin C?(s k+ \ y, w k+ *, z k ; x k ) + \\\y - y k f Sl , 
w k+1 = argmin £%(s k+1 , y k+1 , w, z k -x k ) + ±\\w - w k \\% , 

z k+1 = argmin {s k+1 , y k+1 , w k+1 , z; x k ), 

x k+l = x k + rcr(s k+l + A*y k+1 - Qw k+1 + z k+1 - c). 

Step 2. If a termination criterion is not met, set k := k + 1 and go to Step 1. 

Note that in the above Algorithm sGS-sPADMM, one can always choose S\ = 0 if A : X —x 
is surjective and £2 = 0 if W = Range (Q). The global convergence of Algorithm sGS- 
sPADMM is established in [35] by connecting it into an equivalent sPADMM scheme (1.3a)- 
(1.3c) for solving a particular problem of the form (1.1). By using the same connection, just as 
for the primal case, one can use Theorem 3.2 to derive the linear rate convergence of the infinite 
sequence {(s k ,y k ,w k ,z k ,x k )} generated by Algorithm sGS-sPADMM if Assumptions 2.1 and 
3.1 hold for problem (3.27) and r G (0, (l + \/5)/2). As mentioned earlier, Assumption 3.1 holds 
automatically if 1C is convex polyhedral. However, for a non-polyhedral 1C, there exist few results 
about the existence of the error bound condition as in Assumption 3.1 except for K, to be either 
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a second order cone [5] or an SDP cone [47], where the strong regularity introduced by Robinson 
[40] is characterised in terms of the strong second order sufficient condition and the constraint 
nondegeneracy. The strong regularity provides a sufficient condition for Assumption 3.1 to 
hold. Since the isolated calmness condition given in Definition 2.2 is a much weaker condition 
than the strong regularity, in the next two sections, we shall conduct a thorough study on the 
isolated calmness in the context of composite semi-definite, convex and non-convex, optimization 
problems. The obtained results on the isolated calmness are not only useful for deriving the 
linear rate convergence of the sPADMM but also represent substantial advancements in the 
context of second order variational analysis for conic optimization problems constrained with 
non-polyhedral convex cones. As a final note to this section, we comment that in all the above 
applications, while the linear operator S may take various values, the linear operator T = 0. 


4 Characterizations of the isolated calmness for semi-definite 
optimization problems 


Let Z be a finite dimensional real Euclidean space. For an integer p > 0, let be the positive 
semi-definite cone of all symmetric positive semi-definite matrices in the space § p of p by p real 
symmetric matrices. Denote y := x Z and §]]_ := — = (S+) 0 . Next, we shall consider the 

isolated calmness for the KKT system to the following semi-definite optimization problem: 


min f(x) 
s.t. G(x) £ 1C, 


(4.1) 


where / : X —>• 5R is a twice continuously differentiable function, Q : X —>• y is a twice continu¬ 
ously differentiable mapping with Q = (0, for <f> : X — > and tp : X —> Z, /C = 8^ x R and 
V C Z is a nonempty convex polyhedral set. Let = {x € X : G(x) € 1C} be the feasible set for 
problem (4.1). Let x € <J>. We say that Robinson’s constraint qualification (RCQ) for problem 
(4.1) holds at x if 

0 € int {G(x) + DQ(x)X — 1C}, 

where “int” denotes the topological interior part of a given set. The Largangian function of 
problem (4.1) is defined as 


C(x;y,z) := f(x) + (y,(p(x)) + (z,ip(x)), V (x,y,z) € X x S p x Z. 


For any (y, z) £ § p x Z, let D x £(x ; y, z ) denote the derivative of £(•; y, z) at x £ X and denote 
V x C(x-,y,z) := (D x C(x',y,z))*. If there exists (y,z) £ y such that ( x,y,z ) satisfies the KKT 
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system 


V x £(x; y, z) = 0, 

(y,z) £ Njc(G{x)), 


(4.2) 


then we call x a stationary point of problem (4.1) and (y, z) a Lagrangian multiplier of problem 
(4.1) at x. Here Njc(w) denotes the normal cone of /C at w £ y. Denote by A(x) the set of 
all (y,z) £ § p x Z satisfying (4.2). If x is a local minimizer to problem (4.1), then the set 
A(x) is nonempty, convex and compact if and only if the RCQ holds at x. The strict Robinson 
constraint qualification (SRCQ for short) at x with respect to (y, z) £ A(x) is defined by (see 
Bonnans and Shapiro [ 6 ]) 


Dg(x)A' + r^(e?(x))n(y,^) ± =y, (4.3) 

where for any vector w £ y, w ^ := {y £ y : (w,y) = 0}. Obviously, the SRCQ is more 
restrictive than the RCQ. It follows from Bonnans and Shapiro [ 6 , Proposition 4.50] that the 
set of Lagrange multipliers A(x) is a singleton if the SRCQ (4.3) holds. 

Let x £ <f> be a feasible point. The critical cone of problem (4.1) at x is defined by 


C(x) := {d £ X : D Q(x)d £ T K (S(x)), D f(x)d < 0 }. 

Definition 4.1 (The second-order sufficient optimality condition). Let x be a stationary point 
of problem ( 4 - 1 ) at which A(x) / 0. We say that the second-order sufficient optimality condition 
for problem (4-1) holds at x if 

sup \{d, A7 2 xx C{x] y, z)d) + 2 /y, Dcj)(x)d [— 4>(x)\ ' l D0(x)d\| > 0, V0 ^d£C(x), 

(y,z)eA(x) * 

where for (y, z) £ E> p x Z, V^. x C(-;y, z) := D X [V X £](-; y, z) and for any matrix S € § p , denotes 
the Moore-Penrose pseudo-inverse of S. 


If follows from [ 6 , Theorem 3.86] that if the second-order sufficient optimality condition for 
problem (4.1) holds at x, then the second-order growth condition for problem (4.1) holds at x, 
which implies that x is a strictly local optimal solution to problem (4.1). 

Define the KKT mapping G : X x E? x Z ^ X x W x Z, associated with problem (4.1), by 


G(x,y,z) 


X7 x £(x;y,z) 

-G(x) + U/c(G(x) + (y, z)) 


V (x, y, z) £ X x S p x Z. 


(4.4) 


For characterizing the isolated calmness property for the mapping G 1 , we need some simple 
but useful properties on the non-polyhedral cone Eff and the polyhedral set V. 
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Suppose that A G § p and P G S^. are two matrices satisfying A G IVgp (P) or equivalently 
B G P§p (A) with A € Note that AB = BA = 0 and B = II§p ( B + A). Let C := B + A 
and Ai > A 2 > ... > A p be its eigenvalues being arranged in the non-increasing order. Define 
a := {i : \ > 0, i = 1,... ,p}, f3 := {i : Aj = 0, i = 1,... ,p} and 7 := {i : Aj < 0, i = 1,... ,p}. 
Then there exists an orthogonal matrix P G lR pxp such that 


A a 

0 

1 

0 


1 

O 

P 

0 

1 

0 


A a 

0 

1- 

0 

0 

0 

0 

a, 

II 

tt) 

Eh ~ 

Oh 

0 

0 

0 

P T , C = P 

0 

0 

0 

1 

0 

0 

0 

1_ 


1 

0 

0 

> 

1_ 


1 

0 

0 

1 

c" 

<1 


(4.5) 


where A a >- 0 is the diagonal matrix whose diagonal entries are A, for iGa and A 7 -< 0 is the 
diagonal matrix whose diagonal entries are A j for j G 7, respectively. Write P = [P a Pp P 7 ] 
with P a G 5R px l Q l, P p G K px ^ and P 7 G 5R px W and define T, T G 3?I“I X H by 


T ij = t X \ , r T ij = l-T ij , V (i,j + |a U f3\) G a x 7. 

A ? ; Aj 

It is known from [3, 4] that IIgp (•) is directionally differentiable everywhere and from [48, 38] 
that the directional derivative of IIgp at C along H G § n is explicitly given by 


n(,p(C;P) 


0 0 PjPP 7 oT 

0 (PjHPp) PpHPry 

p 7 t HP a o r T p 7 t HPp p 7 t pp 7 


(4.6) 


where “o” denotes the Hadamard product. Then, by Arnold [1], we know that the tangent cone 
of at B G takes the form of 


TgP_(P) = {H G S p : H = 14 (P; P)} = {P G S p : [P Q P /3 ] T P[P a P^] A 0} 

and the critical cone of § at C, associated with A G IVgp (P), is given by 

CgP_(G) := TgP_(P) n A 1 = {P G § p : PjP[P Q P^] = 0, PjHPg A 0} . (4.7) 

Analogously, the critical cone of at C , associated with B G IVgp (A), is given by 

C K (C) := Tgp (A) n P 1 = {P G § p : P^P^ P 7 ] = 0, PjHPp P 0} . (4.8) 

Lemma 4.1. Suppose that A G S p and P G S!. are two matrices satisfying A G Pgp (P). Let 
A, P and C := P + A have the spectral decompositions as in (f.5). Then we have the following 
results: 
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(i) For any given matrix H G E> p 


H G (C § P_ (C)j° 4=> PiHPy = 0 & H G C s p (C) 

and 

H G (C K (C)J° <=> Pi'HP r = 0 & H G C § p_ (C). 

(ii) Lei AA and A B be two matrices in W. Then 

AA - U' sP (C; AA + AB) = 0 

i/ and only if 

Pl(AA)[P a Pp\=0, 

Pi(AA)P 7 or = Pl(AB)P, O T, 

Pj (AA)Pp = n s w(Pj{AA + A B)P P ), 

[Pp P 7 ] T (AB)P 7 = 0. 

Moreover, the relations in (4-9) imply 

A AgC§p(C) & (AA,AB) = 2(A, (AA^-B^AA)). 

Proof. The conclusions of part (i) follow directly from (4.7) and (4.8) while the conclusions of 
part (ii) can be derived with no difficulty from (4.5), (4.6), (4.7) and the fact that 

pj (A A)Pp = n § ,„ (Pj(AA + A B)Pp) Pj (A A)P P A Pj (A B)P P G S*. 

We omit the details here. □ 

Lemma 4.2. Let V C Z be a given nonempty convex polyhedral set. 

(i) Let a,b,c G Z. Write c + := Ibp(c) and c_ := c — c+. Then 

Ulp(c-b) = U Tr(c+ )nAb). (4-10) 

Moreover, 

a — IIp(c; a + b) = 0 (4.11) 

if and only if 

a G T P (c +) n d: & b G A Tp(c+)nc x(a). (4.12) 
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(ii) Let b G V and 0 G a + N P {b). For the critical cone 


C P (b — a ) := Tp(6) n a^~, 


we have 

C v (b - a) = Sl a , (4.13) 

where Sb A is a nonempty closed convex cone defined by 

Sb,a := {u£ Z : (u, b) + (<5p)'(-a; -u) = 0}. 

Proof. Since V is a nonempty convex polyhedron, we have from Theorem 4.1.1 of [20] that (4.10) 
is true and equality (4.11) is equivalent to 


a = U T v p+) nM a + b )> 

which is equivalent to (4.12). So the conclusions in part (i) hold. 

Now we turn to the proof of part (ii). It follows from [42, Corollary 19.2.1] that is a 
proper closed convex polyhedral function. Then we know from [42, Theorem 23.10] and [42, 
Corollary 23.5.3] that 

(^)'(-a;u) = 4_>), Vu€Z, 

where 

V- a :={b' eV:(-a,b')=5* P (-a)}. 

By using the assumption —a € N-p(b) = 86 -p(b), we know that b G dS p (—a). Therefore, b G V~ a 
and 

Sb,a = {u : {u,b) + (6 P )'(-a;-u) = 0} 

= {u : (u, b) + 5fi_ a (—u) = 0} 

= { u : (u, b) - (u, b') <0, V b' € V- a } 

= {u : (u, b' - b) > 0, V b' G TC a } 

= ~N v _ a (b). 

Thus, Sb, a is a nonempty closed convex cone with Sf a = T-p_ a (b). Since V- a is a polyhedral set 
and V-a = V n L, where L := {b' G Z : ( b' — b,a) = 0}, we have 

Tp_ a {b ) = T v (b) n T L (b) = Tp(b ) n a ± . 


Therefore, 

C v (b-o)=T v _ a (b) = Sl a , 

which shows that (4.13) holds. The proof of this lemma is completed. 


□ 
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Lemma 4.3. Let A : X —» S p x Z be a linear operator and ( 01 , 02 ) G § p x Z. Then v := 
(v \, u 2 ) € § p x Z is a solution to the following system of equations 


A*v 

= 0, 


Ilgp (ci; v \) 

= 0, 

(4.14) 

n p(c 2 ;u 2 ) 

= 0 



if and only if 


v G 


AX + T/c(c+) n ct 


where c + := n^(c) = (II s p (ci), Hp(c 2 )) and c_ = c— c+. 


(4.15) 


Proof. We have from Lemma 4.1 and (4.7) that 

n §p (ci;ui) = 0 vi G [T s p_ ((ci)+) n (ci)f]°. 

Since P is a convex polyhedron, we have from part (i) of Lemma 4.2 that 

~A l 'p(c 2 ',V 2 ) = 0 <*=>■ V 2 G [Tp((c 2 ) + ) PI (C2)f]°. 

Thus, v satisfies (4.14) if and only if 

A*v = 0 & v G [T/c(c+) n c^z]° , 

which is equivalent to saying that (4.15) holds. The proof is completed. □ 

Lemma 4.4. Let x G 4* be a stationary point of problem (4-1) with ( y,z ) G A(x) A 0- Let the 
KKT mapping G be defined by (4-4)- Then G -1 is isolated calm at the origin for ( x,y,z ) if and 
only if (d x , d y , d z ) = 0 for any (d x , d y ,d z ) G X x§ p x2 satisfying G'((x, y, z); (d x ,d y , d z )) = 0. 


Proof. By noting that G is a locally Lipschitz continuous mapping around (x, y, z) and it is 
directionally differentiable at ( x,y,z ), we have for (d x ,d y ,d z ) £4x^x2 that 


DG((x, y, z)\0)(d x ,d y ,d z ) 

( G(x + t k d x ,y + t k d y ,z + t k d z )-G(x,y,z) 

= < Inn - for certain tb \ U 

k — yoo 

= {G'((x, y, z); (d x ,d y , d z ))}. 

Thus, from (2.5), we have for any (d x ,d y ,d z ) € X x § p x Z that 

(1 d x ,d y ,d z ) € DG~ 1 ( 0|(x,y,z))(0) G'((x,y,z ); (d x ,d y ,d z )) = 0, 
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which, together with Lemma 2.1 and the fact that G'((x , y, z); (0,0, 0)) = 0, implies that G 1 is 
isolated calm at the origin for (x, y, z) if and only if 

G ((2, y, zf ( d X t dy , d z f) — 0 ' (d x , dy , d z ) — 0. 

This completes the proof. □ 

Theorem 4.1. Let x € <J> be a stationary point of problem (4-1) with (y, z) € A(x) f 0. Then 
we have the following results: 

(i) If the second-order sufficient optimality condition for problem (4-1) holds at 2 and the SRCQ 

(4-3) holds at 2 with respect to (y, z), then G^ 1 is isolated calm at the origin for (x,y,z ). 

(ii) IfG- 1 is isolated calm at the origin for (x, y, z), then the SRCQ (4-3) holds at x with respect 

to (; y,z ). 

(hi) IfG- 1 is isolated calm at the origin for (2, y, z) and the quadratic form 

q : ( d x , d x ) {d x ,V 2 xx C(x\ y , z)d x ) + 2 <^y, D fi(x)d x [—0(^)] t D fi(x)d x ^ 

satisfies 

q(d xi d x ) >0, Md x <E C(x) & q(d x ,d x ) = 0, d x <E C(x) =>- W 2 xx C(x\y,z)d x = 0, 

then the second-order sufficient optimality condition for problem (4-1) holds at 2. 

Proof. Since (y,z) € A(x), we know y € Ngp (fi{x)) and z € N-p(iJj(x)). Without loss of gener¬ 
ality, we can assume that A := y, B := <f(x) and C := B + A have the spectral decompositions 
as in (4.5). 

We first prove part (i). Let (d x ,d y ,d z ) € X x x Z be arbitrarily chosen such that 
G'((x,y,z);(d x ,d y ,d z )) = 0. Since the SRCQ (4.3) holds at 2 with respect to (y, z), we have 
from [6, Proposition 4.47] that the set of Lagrange multipliers of problem (4.1) at 2 is a singleton, 
namely A(x) = {(y,z)}. In this case, we can write the critical cone C(x) as 

C(x) = C\(x) fl C 2 (x), 

where 

Ci (2) ={d x e X : D c/>(x)d x € T s r (</>(2)), (y, D(j)(x)d x ) = 0}, 

C 2 {x) = {d x € X : Df(x )d x € T-p(^( 2)), (z, Dij}(x)d x ) = 0}. 
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Since G'((x,y,z ); (d x ,d y ,d z )) = 0, we have 


Vl x £(x;y,z)d x + Dg(x)*(dy,d z ) =0, 

(4.16) 

-D Q(x)d x + Ii' K (Q(x) + (y, z)\ D Q(x)d x + (d y , d z )) = 0. 

The second equation in (4.16) can be split into 

D(j)(x)d x - Ilgp (cj)(x) + y; D <f)(x)d x + d y ) = 0, 

Dip(x)d x — II tp(ip(x) + z; T>ijj(x)d x + d z ) = 0. 

Thus, we know from part (ii) of Lemma 4.1 that 

D 4>{x)d x € T § r (4>(x)) n y 1 & (D<p(x)d x ,d y ) = 2 (y, D (j)(x)d x [-0(x)] f D <j>(x)d x ) 
and from (i) of Lemma 4.2 that 

D ip(x)d x € Tp(ij}(x)) ilz 1 & (d z , D tp(x)d x ) = 0. 


Therefore, d x E C(x). By taking the inner product between d x and both sides of the first 
equation in (4.16), we obtain 

(d x ,V 2 xx C(x,y,z)d x ) + (d x , DQ(x)*(d y , d z )) = 0 

and thus 

(d x , V 2 xx C(x, y, z)d x ) + 2 D c/)(x)d x [-^(x)] f D <j)(x)d x ^ = 0. 

It then follows from the second-order sufficient optimality condition for problem (4.1) at x that 
d x = 0. Hence (4.16) is reduced to 


~DQ(x)*(d y ,d z ) 

= 0 , 

n^ P (4>(x) + y; d y ) 

= 0 , 

W v (ip(x) + z; d z ) 

= 0 . 


In view of Lemma 4.3, we obtain 


(d y , d z ) € DG(x)X + Tic(G(x)) n (y, z) 


which implies ( d y ,d z ) = 0 from the assumed SRCQ (4.3). Therefore, ( d x ,d y ,d z ) = 0. Then, we 
know from Lemma 4.4 that G is isolated calm at the origin for ( x,y,z ). 

Now we prove part (ii). Suppose that the SRCQ (4.3) does not hold at x for (y,z) E A(x), 
namely 

r := D Q(x)X + T K (Q(x)) n (y, z) 1 - ± y. 
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Then there exists 0 7 ^ (y, z) € § p x Z such that (y, z) € T° or equivalently 


0 / (y, z) € (d^x)*) 1 - n [(T SP _ (cf(x)) n y x )° x (7>(t/,(x)) n ^)° 
Then we have from Lemma 4.3 that 


D G{x)*(y,z) =0, 

n sr (<K^) + y; y) =o, 

IIp(^(x) + 2 ; 2) = 0, 


which imply 


G'((x, y, z); (0 ,y,z)) = 0, 


that is 

0 € DG((x,y,z)\0)(0,y,z). 

Since G _1 is assumed to be isolated calm at the origin for ( x,y,z ), we obtain from Lemma 2.1 
that (y, z) = 0. This contradiction shows that the assertion in part (ii) is true. 

Finally, we prove part (iii) by contradiction. Suppose that the second-order sufficient opti¬ 
mality condition for problem (4.1) does not hold at x. Since G _1 is assumed to be isolated calm at 
the origin for (x, y, z), we have A(x) = {(y , z)}. Thus, there exists a vector 0 / d x € C(x) satisfy¬ 
ing q(d x ,d x ) = 0. We then know from the conditions given in part (iii) that V xx £(x; y, z)d x = 0 
and thus (y, D (f>{x)d x [— 4>(x)}^ D cf)(x)d x } = 0. Moreover, from the definition of C(x) and Lemmas 
4.1 and 4.2, we have 

D c/)(x)d x - II' P (cj)(x) + y; D (j)(x)d x ) = 0, 

(4.17) 

~D^>(x)d x — Hp(ip(x) + z; Dip(x)d x ) =0. 

By using V xx £(x;y,z)d x = 0, (4.17) and the expression of the directional derivative of G at 
( x,y,z ), we get G'((x,y,z)‘,(d x , 0,0)) = 0 with d x / 0. Then, by Lemma 4.4, we arrive at a 
contradiction with the isolated calmness of G^ 1 at the origin for ( x,y,z ). Therefore, we must 
have q(d x ,d x ) > 0 for d x € C(x ) \ {0}. That is, the second-order sufficient optimality condition 
for problem (4.1) holds at x. The proof is completed. □ 


Based on Theorem 4.1, for linearly constrained convex optimization problems, we obtain the 
following complete characterization on the isolated calmness of G _1 . 

Corollary 4.1. Let f be a twice continuously differentiable convex function, Q be an affine 
mapping and x be a minimizer to problem (4-1) with A(x) f 0. Then Gis isolated calm at 
the origin for (. x , y, z) with (y, z) € A(x) if and only if the second-order sufficient optimality 
condition for problem (4-1) holds at x and the SRCQ (4-3) holds at x for ( y,z ) € A(x). 
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5 Convex composite quadratic semi-definite programming 


In this section we shall further study the isolated calmness for the following important convex 
composite quadratic SDP: 

min -(x, Qx) + (c,x) 
s.t. Ax = b, x e §+ n V, 

where c G § p , b € Q : —>• is a self-adjoint positive semi-definite linear operator, 

A : § p —>• is a linear operator and P is a simple nonempty convex polyhedral set in § p . As 

in Subsection 3.1, by introducing an additional variable u G E> p , we can rewrite problem (5.1) 
equivalently as 


min ~(x,Qx) + (c,x) 

s.t. Ax = b, x — u = 0, x € §+, u€V. 

Suppose that (x,u) € x V is an optimal solution to the convex optimization problem (5.2). 
Note that u = x. Let A p(x,u), which may be an empty set, denote the set of Lagrange 
multipliers (s, y, z, v) G x x S p x § p for problem (5.2) at (x, u ) such that (x, u, s, y, z, v ) 
satisfies the following KKT system 


Qx + c — A*y — z — s = 0, z + v = 0, 
b — Ax = 0, u — x = 0, s€ N§p (~x), v G N-p{u). 


(5.3) 


The KKT mapping Gp , associated with problem (5.2), for any (x,u, s,y, z,v) x S p x$ p x 
sftm x S p x is given by 


G P (x,u,s,y,z,v) 


Qx — A*y — z — s + c 

Z + V 

x + n § p (—x + s) 
Ax — b 
x — u 

—u + II p(u + v ) 


(5.4) 


We also define the reduced KKT mapping Fp. associated with problem (5.2), as follows: for any 
(x, u, y , z) G x x x § p , 


Fp(x, u, y, z) 


x + n§p (—x + Qx — A*y — z + c) 
— 14 + ILp(t4 — z) 

Ax — b 
x — u 


(5.5) 


By using Lemma 4.4, we can easily obtain the following equivalence on the isolated calmness 
property of ( Gp )~ x and (Fp) -1 . 
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Proposition 5.1. Let ( x,u,s,y,z,v ) £ § p x § p x § p x x § p x § p be a solution to the KKT 
system (5.3). Then ( Gp) _1 is isolated calm at the origin with respect to ( x,u,s,y,z,v ) if and 
only if (Fp)^ 1 is isolated calm at the origin with respect to (x,u,y,z ). 

The critical cone of problem (5.2) at (x, u) is given by 

C(x, u) = {( d x ,d u ) € § p x§ p : Ad x = 0, d u -d x = 0, d x £ T s p (x), d u £ T v (u), (Qx+c,d x ) = 0}. 

If Ap(x, ft) A then for any (s, y, 2, n) € Ap(x, ft), 

C{x,u) = {(d x ,du) £ § p x § p : AZ X = 0, = 0, d x £ C§p (x - s), € Cp(?Z - z)}. (5.6) 

The Lagrange dual of problem (5.2) takes the form of 

max inf { -(x, Qx) + (v, x) \ + ( b , y) - 6p(-z) 

xesr 1 2 J (5.7) 

s.t. s + A*y + v + z = c, s € §+, 

which is equivalent to 

max (b,y) - -(w, Qw) - 8p(-z) 

s.t. s + A*y — Qw + z = c, (5.8) 

s £ § p , w € W, 

where W is any linear subspace in § p that contains Range Q, e.g., VV = § p or W = Range Q. 
By introducing an additional variable t, we can reformulate problem (5.8) equivalently as 

max (b, y) - ^(w, Qw) - t 

s.t. s + A*y — Qw + z = c, (5.9) 

s € § p , w € W, (z, t ) € epi 9 , 

where 

9(z) := S^(-z), Vz€ S p . 

Let ( s,y,w,z ) € S 1 * x x W x § p be an optimal solution to problem (5.8). Then, obviously, 
(s, y, w, z, 6(z)) is an optimal solution to problem (5.9). We use A p>(s, y, w, z) to denote the cor¬ 
responding set of Lagrange multipliers for problem (5.8) at (s, y, w, z), that is x £ A p>(s, y. w, z) 
if and only if (s, y, w, z, x) satisfies the following KKT system 

0 £ x + iVgp (s), Ax — b = 0, Qw — Qx = 0, 0 £x + dd(z), c —s — A*y + Qw — z = 0. (5.10) 
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Thus, the KKT mapping Fp, associated with problem (5.8), can be defined for any (s, y, w, z, x) € 
x r x W x x §P that 


F D (s,y,w,z,x) 


Ax - b 
Qw - Qx 

—s — A*y + Qw — z + c 
s + II § p (—s + x) 

—z + Pr o(z — x) 


(5.11) 


Note that for any x £ A r>(s,y,w, z), it holds that 

0 £ x + 89(z) 0 € (x, 1) + N epig (z, 9(z)) (z, 9(z)) = n e p ie ((^, 9(z)) - (.x , 1)). 

Moreover, since 0 : —>• (—oo, +oo] is a proper closed convex polyhedral function [42, Corollary 

19.2.1], we know from convex analysis [42, Theorem 23.10] that 

T ep ie(z,9(z)) = ( N epig (z,9(z)))° = {( u,t ) £ x 9ft : 9'(z;u ) < t}. 

Thus, for any x £ A jo(s,y,w,z), 

T e pie(z, 9(z)) n (x, I ) -1 = {(u,t) £ S p x : t = (u,—x) = 9'(z;u)} 

= {(u, t) € § p x 3ft : u € S x>z , t = (u , — x)}, 

where for any ( x,z ) £ S p x S p , the set S x>z is defined by 

S x , z ■= {u £ § p : (u, x) + 9'(z ; u) = 0} = {u £ E> p : (u , x) + (Sp)'(-z; —u ) = 0}. (5-12) 

Lemma 5.1. Let x £ A £>(s, y, w, z) / 0. T/ien /or any (Sz, St) £ § p x 3ft and Ac £ § p , 

(Sz, <5t) = (n e p i £)) / ((z — x , 9(z) — 1); (As — <5x, <5t)) < 5,2 = (Pr g )'(z — x;Sz — Sx), St = (Sz, —x). 

Proof. By using Lemma 4.2, we have 

(Sz, St) = (n e pi 0 )'((2 - x, 9(z) - 1); (Sz - Sx, St)) -<=>- -Sx £ Ns x ^(Sz), St = (Sz, -x). 

By noting that for any v £ § p , Prg(v) = v + II -p(-v), we know from Lemma 4.2 that 
Sz = (Pr e)'(z — x;Sz — Sx) Sz = (Sz — Sx) + IIp(x — z;Sx — Sz) •<=> —Sz € N$* _(<5x). 
The conclusion of this lemma then follows. □ 
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The KKT mapping Gp, associated with problem (5.9), for any (s,y,w,(z,t),x,u,(v,Q) € 
§ p x W n x W X (§ p X ») X § p x $P x (SP x £) is given by 


G d (s, y, w, ( z,t),x,u , (n,C)) 


x — u 
Ax — b 
Qw — Qx 

(*,1) + (v,C) 

—s — Vl*y + Qw — z + c 

S + IlgP (— S + It) 

~(z,t) +U cpi g((z,t) + (V,C)) 


(5.13) 


By using Lemmas 4.2, 4.4 and 5.1, we can obtain with no difficulty the following equivalence on 
the isolated calmness property of (Go) -1 and (Fp>)~ 1 . 


Proposition 5.2. Let (s, y. w, (z, 0(z)),x, u,(v,— 1)) € § p x$ft m xWx (§ p x !R) x § p x x (§ p x$R) 
6 e suc/t that GD(s,y,w,(z,6(z)),x,u,(v,— 1)) = 0. Then (Gd) _1 is isolated calm at the origin 
with respect to ( s,y,w , (z, 9(z)), x, u, (v,— 1)) if and only if (Pd) -1 is isolated calm at the origin 
with respect to ( s,y,w,z,x ). 


Based on the equivalence between problem (5.9) and problem (5.8), as in [52] for the linear 
SDP case, we can now introduce the concept of the extended SRCQ for problem (5.8) in the 
following definition. 


Definition 5.1. Suppose that Ap(x,u) A 0- We say that the extended SRCQ for the dual 
problem (5.8) holds at A (x,u) with respect to (x, u) if 


conv 


U (^ S + ^ ^ X± Sx,Z^ 

„ (s,y,z,v)eAp(x,u) 


+ A*M m -QW = § p , 


(5.14) 


where “conv” denotes the convex hull of a set. 


Now we can establish the relationship between the second-order sufficient optimality condi¬ 
tion for problem (5.2) and the extended SRCQ for problem (5.8). 

Proposition 5.3. Let(x,u) £S+xP be an optimal solution to problem (5.2) with Ap(x,u) A 0- 
Let W C § p be any linear subspace that contains Range Q. Then the following two conditions 
are equivalent: 

(i) The second-order sufficient optimality condition for the primal problem (5.2) holds at (x, u): 

sup i(Qd x ,d x ) + 2(s,d x x^d x )\ > 0, VO / (d x , d u ) € C(x, u). (5.15) 

( s,y,z,v)£Ap(x,u ) ^ J 
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(ii) The extended SRCQ (5.14) for the dual problem (5.8) holds at A p(x,u) with respect to 
(x,u). 

Proof. For notational convenience, denote 

T := conv ! [J (t s p + (s) fliH l and V := T + A*iR m - QW. 

((s,i;,z,»)eAp(x,u) J 

“(i) => (ii)” We prove this part by contradiction. Suppose that the extended SRCQ (5.14) 
for the dual problem (5.8) does not hold at A (x,u) with respect to (x,u). Then V A BP. Let 
cl(P) denote the closure of V. Since cl(D) =4 § p (cf. [42, Theorem 6.3]), there exists a point 
a G EP but a ^ cl(D). Let h := II c i(x>)(a) — a. By using the fact that cl (D) is a closed convex 
cone in S p , we have 

(M) > 0, VdGcl(P), 

which, together with the assumption Range Q C W, implies that Ah = 0, Qh = 0 and 

(h,d) > 0, VdeT. (5.16) 

Let ( s,y,z,v) be an arbitrary point in A p(x,u). Then 0 € s + A/§p (x) and 0 G z + N-p(u). Since 
x G iVgP (—s), without loss of generality, we can assume that A := x, B := —s and C := —s + x 
have the spectral decompositions as in (4.5). Then, by using (5.16), part (i) of Lemma 4.1 
(applying to A = x and B = —s and using (s) = —T§p (— s)) and part (ii) of Lemma 4.2 
(applying to a = z and b = x), we obtain (recall that x = u) 

h € Cgp (x — s), (s, hxfh) = 0 Sz h G Cp(x — z). 

Therefore, 0 A (^-> h) € C(x,u). Thus, by using the condition (5.15), we know that there exists 
(s, y, z, v) G A p(x, u ) such that 

(Qh, h) + 2 (s, hxfh) > 0, 

which contradicts the proven Qh = 0 and (s, hx^h) = 0. This contradiction shows that this part 
holds. 

“(ii) =>- (i)” For the sake of contradiction we suppose that the second-order sufficient 
optimality condition (5.15) for the primal problem (5.2) at (x,u) fails to hold. Then there exists 
0 A (h, h) G C(x, u) such that 

sup i(Qh,h) + 2(s,hx^h)\ = 0, 

(s,y,z,v)€.Ap(x,u) ^ J 
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which implies 


(Qh,h) = 0 and (s,hx^h) = 0 , V (s,y, z,v) £ Ap(x,u). 

Let (s, y, z, v ) be an arbitrary point in A p(x, u). By using the fact that 0 £ s+N§p (x) if and only 
if x € N§p (—s), without loss of generality, we can assume A := x, B := —s and C := —s + x have 
the spectral decompositions as in (4.5). Then, from (s,hx^h) = 0 we know that P^hP^ = 0. 

Since the extended SRCQ (5.14) is assumed to hold, there exist y £ W l , w £ W and d € T 
such that —h = d + A*y — Qw. By Caratheodory’s theorem, there exist a positive integer 
k < p(jp + l )/2 + 1 , scalars //* > 0 , i = 1 ,,.., k, with + ^2 + ■ ■ ■ + yk = 1 , and points 

d l £ IJ (t s p + (s) Hi 1 ! Sx , z ) , i = 1, ■ • • , k 

(s,y,z,v)eA P (x,u) 

such that d = d 1 + 112 d 2 + ... + pkd k ■ For each d l , there exist (s*, y l , z l , v l ) € A p(x,u), 
d\ £ T § p (s*) n x 1 - and d\ £ S S z i such that d 1 = d\ + d\- Then, by using Qh = 0, P^hP y = 0, 
(h , h) £ C(x, u), T s p (s) = — T s p (—s), part (i) of Lemma 4.1 and part (ii) of Lemma 4.2, we have 

k 

(h, h) = (-d - A*y + Qw, h) = {-d, h) = - + 4 ty < 0 . 

i —1 

This contradiction shows that this part is also true. □ 

If A p{x,u) is a singleton, we have the following corollary. 

Corollary 5.1. Let (. x,u ) £ x V be an optimal solution to problem (5.2). If Ap(x, u) = 
{( s,y,z,v )}, then the following two conditions are equivalent: 

(i) The second-order sufficient optimality condition for the primal problem (5.2) holds at (x,u): 

(Qd x ,d x ) +2(s,d x x^d x ) > 0, V0 A (d x , d u ) £ C(x, u). (5-17) 

(ii) The SRCQ for the dual problem (5.8) holds at ( s,y,z,v) with respect to (x,u): 

% (a) Hi 1 ! S x - Z + A*iR m - QW = § p . (5.18) 

In the next proposition, we shall establish an analogous result to Proposition 5.3 between 
the second order sufficient optimization condition for the dual problem (5.8) with W = Range Q 
and the extended SRCQ condition for the primal problem (5.2). 

Proposition 5.4. LetW = Range Q and (s,y,w,z) £ § p x3ft m xWx§ p be an optimal solution to 
the dual problem (5.8) with Af)(s,y,w, z) A 0- Then the following two conditions are equivalent: 
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(i) The second-order sufficient optimality condition for the dual problem (5.8) holds at (s, y, w, z ) : 

sup \(Qd w ,d w ) + 2(x,d s s t (i s )| > 0, VO f (d s , d y , d w , d z ) € C(s, y, w, z), (5.19) 

16A D (s,y,w,z) ^ J 

where C(s,y,w,z ) is the critical cone consisting of all the vectors (d s , d y , d w , d z ) G § p x 
x5i m x W x § p such that 

d s + A*d y - Qd w + d z = 0, d s € T s p (s) Hi 1 & d z G S Xt g. 


(ii) The extended SRCQ for the primal problem (5.2) holds at Ajj(s,y,w, z ) with respect to 

(s,y,w,z) ■ 



“(i) ==> (ii)” Suppose that (5.20) does not hold. Then, by using the similar arguments as 
in the first part of the proof for Proposition 5.3, we know that there exists 0 f h = (Ji\. h 2 ) G 
x such that 


(h, d) > 0 , M d G r, 


which implies that for any x G A £>(s,y,w, z), 

A*h\ + h 2 G (x) n & h 2 € ^Tp(i) fl . (5.21) 


Let x G A D (s, y. w, z) be fixed but arbitrarily chosen. Then 0 G x+N§p{s) and 0 G x+d6{z). 
Since 0 G x + N§p (s) if and only if x G Ngp (— s), we can assume that A := x, B := — s and 
C := — s + x have the spectral decompositions as in (4.5). Then we know from (5.21), part (i) 
of Lemma 4.1 and part (ii) of Lemma 4.2 that 

Pj (A*h 1 + h 2 ) P 7 = 0, A* hi + h 2 € ~(C s r_ (-s + *)) = T s p (s) & h 2 G S x - Z . 

Let d s = —(A*hi + h 2 ), d w = 0 G W, d y = hi and d z = h 2 . Then we have 

0 (d 8 , d yi d w , d z ) G C(s , y , w , z') & ( x , d s s^ d s') — 0, 

which contradicts (5.19). This completes the proof of (i) ==> (ii). 
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“(ii) ==> (i)” For the sake of contradiction suppose that the second-order sufficient optimality 
condition (5.19) for the dual problem (5.8) at ( s,w,y,z ) does not hold. Then there exists 
0 ^ ( d s ,d y , d w ,d z ) G C(s, y, w, z) such that 

sup UQd w ,d w ) + 2(x,d s s^d s )\ = 0, 

ieeA D(s,y,w,z) ^ J 


which implies 


(Qd w ,d w ) = 0 & (x, d s s'd s ) = 0, VxGA D (s,y,w,z). 


By using the fact that d w G Range Q, we know that d w = 0. Then, by mimicking the proof for 
the second part of Proposition 5.3, we can show that d s = 0, d y = 0 and d z = 0 and reach a 
contradiction to complete the proof of this part. The details are omitted here. □ 


If A £>(s,w,y, z) happens to be a singleton, we have the following corollary. 

Corollary 5.2. Let W = Range Q and (s, y, w, z) G § p x W x x § p 6 e an optimal solution 
to the dual problem (5.8) with Ap>(s,y,w, z) = {x}. Then the following two conditions are 
equivalent. 


(i) The second-order sufficient optimality condition for the dual problem (5.8) holds at (s, y, w, z) : 

(Qd w ,d w ) + 2(x,d s s^d s ) > 0, V 0 7 ^ (d s , d y , d w ,d z ) G C(s, y, w, z). (5.22) 


(ii) The SRCQ for the primal problem (5.2) holds at x with respect to ( s,y,w,z ) : 

j Tgp (s) ns 1 ! {0} x (t v {x) n z X § p . (5.23) 

By noting that (x, u, s, y, z, v) is a solution to the KKT system (5.3), i.e., Gp(x, u, s, y, z, v) = 
0, if and only if Fjj(s, y, w, z, x) = 0 for some w G W such that Qw = Qx, we can now state our 
main theorem of this section. 

Theorem 5.1. Suppose that W = Range Q. Let (x, u, s, y, z, v) G ST x EP x § p x x § p x ET 
be such that Gp(x,u,s,y,z,v ) = 0 and w be the unique point in W such that Qx = Qn). T/ien 
t/ie following statements are equivalent to each other: 

(i) The second order sufficient optimality condition (5.17) for the primal problem (5.2) holds 
at ( x,u ) and the second order sufficient optimality condition (5.22) for the dual problem 
(5.8) holds at ( s,y,w,z ). 
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(ii) The SRCQ condition (5.23) for the primal problem (5.2) holds at x with respect to (s, y. w, z) 

and the SRCQ condition (5.18) for the dual problem (5.8) holds at (s,y,z,v) with respect 
to (x, u). 

(iii) ( G P ) _1 is isolated calm at the origin with respect to (x,u, s,y,w, z,v). 

(iv) (Go ) -1 is isolated calm at the origin with respect to (s,y,w,(z,9(z)),x,u,(—x,— 1)). 

(v) (Fp )” 1 is isolated calm at the origin with respect to ( x,u,y,z ). 

(vi) (Fd ) -1 is isolated calm at the origin with respect to ( s,y,w,z,x ). 

(vii) The second order sufficient optimality condition (5.17) for the primal problem (5.2) holds 
at (x,u) and the SRCQ condition (5.23) for the primal problem (5.2) holds at x with 
respect to ( s,y,w,z ). 

(viii) The second order sufficient optimality condition (5.22) for the dual problem (5.8) holds 
at ( s,y,w,z ) and the SRCQ condition (5.18) for the dual problem (5.8) holds at ( s,y,w,z) 
with respect to (x, u). 

Proof. By using the fact that the conditions in either (i) or (ii) or (vii) or (viii) imply that 
A p(x,u) = {(s,y, z,v)} and Ap>(s, y, w,z) = {x}, we obtain from Corollaries 5.1 and 5.2 that 

(i) (ii) <*=*. (vii) <*=*- (viii). 

By using Lemma 4.2 and the assumption that W = Range Q. we can obtain (v)<^=> (vi) (refer 
to the proof of Lemma 5.1). Thus, by further using Propositions 5.1 and 5.2, we have that the 
statements (iii)-(vi) are all equivalent to each other. Finally, by noting from Corollary 4.1 that 
(iii) (vii), we complete the proof. □ 

Recall that in Theorem 3.2 for the linear convergence rate of the sPADMM, we need As¬ 
sumption 3.1. This assumption holds for problem (5.2) and its dual (5.8) if any one of the eight 
statements in Theorem 5.1 is satisfied. Although Theorem 5.1 is only developed for convex com¬ 
posite quadratic SDP, it is possible to extend it to other convex conic optimization problems 
with the positive semi-definite cone being replaced by some other non-polyhedral but nice cones 
such as the second order cone or any finite Cartesian product of the second order cones and the 
positive semi-definite cones. 
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6 Conclusions 


In this paper, we have provided a roadmap for the linear rate convergence of the sPADMM 
for solving linearly constrained convex composite optimization problems. One significant fea¬ 
ture of our approach relies on neither the strong convexity nor the strict complementarity. Our 
linear rate convergence analysis for the convex composite quadratic programming is quite com¬ 
plete while significant progress in convex nonlinear semi-definite programming, in particular in 
convex composite quadratic semi-definite programming, has been achieved. Perhaps, the most 
important issue left unanswered is to provide error bound results under weaker conditions for 
(convex) composite optimization problems with non-polyhedral cone constraints. Another im¬ 
portant issue is to develop similar results for the inexact version of the sPADMM, which is often 
more useful in practice. However, given the recent progress made on the inexact symmetric 
Gauss-Seidel based sPADMM in [10], it does not seem to be difficult to extend our analysis to 
the inexact sPADMM. 
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